System and method for categorically scoring electronic documents

ABSTRACT

Methods and apparatus are provided for associating scores with search results obtained from a search engine so that the scores can be displayed with the search results. A search of electronic documents, such as webpages, is requested and a set of search results obtained. Each search result in the set references an electronic document. Using the search results, a database containing scores determined for electronic documents is searched to determine whether scores have been determined for the electronic documents referenced by the search results. The scores stored in the database are based on people&#39;s ratings of the electronic document corresponding to the score. If a score has been determined for an electronic document referenced by one of the search results, that score is associated with the search result and the search results are then displayed with any scores associated with the search results.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 12/734,505, filed on May 6, 2010 and titled “METHOD OFDISPLAYING A SUBJECTIVE SCORE WITH SEARCH ENGINE RESULTS”, which is anational entry stage filing of PCT/CA2007/002332, filed on Dec. 20, 2007and titled “METHOD OF DISPLAYING A SUBJECTIVE SCORE WITH SEARCH ENGINERESULTS”, which claims the benefit of, and priority to U.S. provisionalpatent application 60/870,882, filed on Dec. 20, 2006, and is also acontinuation-in-part of U.S. patent application Ser. No. 12/734,504,filed on May 6, 2010 and titled “METHOD AND APPARATUS FOR SCORINGELECTRONIC DOCUMENTS” which is a national entry stage filing ofPCT/CA2007/002330, filed on Dec. 20, 2007 and titled “METHOD ANDAPPARATUS FOR SCORING ELECTRONIC DOCUMENTS”, which claims the benefitof, and priority to U.S. provisional patent application 60/870,882,filed on Dec. 20, 2006, the entire specifications of each of which areincorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION

1. Field of the Art

This invention is in the field of information retrieval systems and inparticular systems and methods for providing an independent scoring ofelectronic documents.

2. Discussion of the State of the Art

The World Wide Web provides a large collection of interconnected contentin the form of electronic documents, images and other media content.Over the years the web has grown to an immense size and containswebpages and other content on just about every subject a person couldthink of. As a result of its growth to such an immense size, locatingcontent on the web has become of primary concern with the result thatthere are 15 numerous search services now available.

Many of these search services take the form of a search engine, where auser can input a search query in the form of one or more search termswith connectors placed in between the terms. The search engine thentakes the search query and attempts to match it to webpages on the webthat have been indexed by the search engine. By matching the searchquery to a number of different webpages, the search engine generates alist of search results and returns the list of search results to theuser. Each search result in the list typically includes a link that auser can use to access the located webpage or other located electronicdocument.

The search engines typically locate what they consider to be “relevant”webpages by using specially created indexes and/or databases where therelevancy of a document identified in an index or database is based onterms from the search query being present. The located documents arethen further ranked so that the “best” results appear higher in the listof search results and the “poorer” results appear closer to the bottomor end of the list of search results.

Additionally, it has become a fairly big business to consult on websitedesign in order to use tricks and loopholes in the more commonalgorithms used by search engines to have a webpage ranked higher insearch results than another webpage which might be as good qualitativelyif not better than the higher ranked webpage.

The ranking of the located search results is typically done usingalgorithms that often base the ranking on how closely the search querymatches the located webpages (usually on how the webpage is described inthe search engine's index or database) and other criteria. In somecases, because the search engines only receive a search query containingsearch terms, the ranking of the webpages located by a search engine canbe heavily based on the occurrence of the search terms in the index ordatabase identifying the webpage, however, other factors can also betaken into account, such as whether the domain name matches the searchquery or whether a webpage is a sponsored link that has paid the searchengine to be ranked higher.

While many of these algorithms may be good at ranking located webpagesby the criteria of “relevancy” used by the algorithms, this ranking isbased on objective factors. They are typically unable to determine whichof the located webpages may be qualitatively “better” than other locatedwebpages, which is often a subjective quality assessment that cannot beassessed on a purely objective basis. By relying on objectively definedparameters such as the number of times a search term appears on awebpage or whether the domain name contains one or more of the terms inthe search query these algorithms fail to provide rankings of thelocated webpages based on the subjective quality of a webpage. Often,even though a webpage may use commonly used search terms and thereforetypically rank quite highly in a list of search results, the overallquality of the webpage may not be that high or as good as another sitethat does not use the search terms as frequently.

While many search engines do not even attempt to address howqualitatively good search results may be, some search engines do usealgorithms that attempt to determine which search results arequalitatively “better” than other search results. One example of this isthe algorithm disclosed by U.S. Pat. No. 6,285,999 to Page, that usesthe number of links between webpages to try to assess the quality of awebpage. The algorithm is based on the underlying theory that websitesthat are linked to by a relatively large number of other unrelatedwebsites are more likely to be qualitatively “better” than websites thathave few other websites linking to it. Even in trying to determine howsubjectively “good” a website might be, this algorithm is still limitedto using objectively measurable factors (in this case the amount oflinks) to attempt to approximate how subjectively “good” a webpage maybe.

However, while the conventional search engines may not be able toproduce subjectively ranked results, conventional search engines such asGoogle™, Yahoo! Search™, Excite™, etc., have massive resources investedin their infrastructures and equipment including the compilation ofmassive indexes of webpages and other electronic documents. Hours andhours of effort and incredible amounts of funding and research have wentinto the creation of these search engines and their search engineindexes. In addition, because of the ever changing nature of theInternet and in particular the World Wide Web these search indexes mustbe constantly maintained and updated. Operating and maintaining apopular search engine is a huge undertaking. A popular search enginemight index hundreds of millions of webpages and respond to millions andmillions of search queries a day. To start from scratch and create asearch engine for the World Wide Web or other parts of the internet is ahuge undertaking that will also have very savvy and well-fundedcompetition from other search engines. While conventional search enginesmay not allow much subjectivity to factor into their ranking of searchresults, their searches do encompass huge amounts of webpages that arecompiled in massive search indexes.

There is a need to provide some type of subjective rating of thequality, popularity or other criteria for search results; a rating thatreflects how “good” a webpage or other electronic document may be, whilestill being able to use the search indexes and massive searching abilityof conventional search engines.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a method andapparatus that overcomes problems in the prior art.

In a first aspect, a method for displaying a score with a search resultis provided. The method comprises: requesting a search from a searchengine using a search query having at least one search term; receiving aset of search results from the search engine in response to the searchquery, each search result referencing an electronic document, the set ofsearch results in an order based on a ranking of the search results bythe search engine; displaying the set of ranked search results in theorder; and displaying at least one score along with the set of searchresults, each score associated with an electronic document referenced byone of the search results and displayed with the one of the searchresults. Each score is determined independently from the ranking of thesearch results by the search engine.

In a second aspect, a data processing system for displaying a score witha search result is provided. The data processing system comprises: atleast one processor; a memory operatively coupled to the at least oneprocessor; a display device operative to display data; and a programmodule stored in the memory and operative for providing instructions tothe at least one processor, the at least one processor responsive to theinstructions of the program module. The program module is operative for:requesting a search from a search engine using a search query having atleast one search term; receiving a set of search results from the searchengine in response to the search query, each search result referencingan electronic document, the set of search results in an order based on aranking of the search results by the search engine; displaying the setof search results in the order on the display device; and displaying atleast one score along with the set of search results on the displaydevice, each score associated with an electronic document referenced byone of the search results and displayed with the one of the searchresults. Each score is determined independently from the ranking of thesearch results by the search engine.

In a third aspect, a method of associating scores determined forelectronic documents with search results obtained from a search enginethat reference the electronic documents is provided. The methodcomprises: providing a database storing scores determined for electronicdocuments; obtaining a set of search results, the set of search resultsobtained from a search engine using a search query having at least onesearch term, each search result referencing an electronic document; andfor each search result in the set of search results, searching thedatabase for any score determined for the electronic document referencedby the search result and if a score has been determined for theelectronic document referenced by the search result obtaining the scorevalue and associating it with the search result. The database isindependent from the search engine.

Online communities in the form of social networks have become popular onthe internet. Online social networks allow members to interact and makeconnections with other members in the online community; either byjoining them together in subgroups or connecting members directly aspeers. Common examples of these online social networks includeFriendster™ and MySpace™, which allow users to link to other users,share information about themselves and send messages to each other.These existing online social networks are primarily focused on membersmeeting other members through linking to pages the members create,bulletin boards or direct messaging services. Often these sites arefocused on geographical locations so that people online can meet peoplewho live near them.

By structuring an online social network so that the members of thesocial network can rate electronic documents such as webpages and otherelectronic content, a subjective rating of an electronic document can beobtained. Because these members form an online community, the onlinecommunity can be structured so that not only can the members rateelectronic documents, but the members in the online network can alsorank other members. In this manner, not only can subjective ratings ofelectronic documents such as webpages or other electronic content begiven a weighted rating, based on a subjective quality of how good anelectronic document is, in the opinions of the members in the onlinecommunity, but the member's ratings of the electronic document can befurther weighted based on how the online community views the members whoare doing the rating. A member that is highly rated by other members inthe online community will have more weight placed on his or her ratingsof an electronic document than another member that is rated much lowerby the online community.

Rather than attempting to indirectly approximate the subjective qualityof an electronic document such as a webpage, by looking at objectivelymeasurable attributes that a computer system can measure and evaluate,the present apparatus and methods allow a rating to be determined forelectronic content that is directly based on the subject qualityassessment of the electronic content by an online community.

In one aspect, a method and apparatus is provided that allowssubjectively determined scores to be associated with search results thathave been obtained from a search engine that have already been ranked bythe search engine using its own ranking algorithm. A search ofelectronic documents, such as webpages, is requested from a searchengine and then a set of search results is obtained from the searchengine. This allows a conventional search engine to be used and theextensive searching capabilities of these conventional establishedsearch engines to be utilized. Each search result in the set referencesan electronic document. Using the search results, a database containingscores determined for electronic documents is searched to determinewhether scores have been determined for the electronic documentsreferenced by the search results. The scores stored in the database arebased on subjective ratings of the electronic document corresponding tothe score. If a score has been determined for an electronic documentreferenced by one of the search results, that score is associated withthe search result and the search results are then displayed with anyscores associated with the search results.

In another aspect. a method for assigning a score to an electronicdocument is provided. The method comprises: assigning a weight factor toeach member of a plurality of members; receiving a rating value of anelectronic document from each of at least two of the members of theplurality of members; and determining a score for the electronic 20document. the score based on each rating value assigned to theelectronic document by a member with each rating value being weighted bythe weight factor assigned to the member who provided the rating.

In another aspect, a computer readable memory having recorded thereonstatements and instructions for execution by a data processing system tocarry out the method of assigning a score to an electronic document isprovided. The method comprises: assigning a weight factor to each memberof a plurality of members; receiving a rating value of an electronicdocument from each of at least two of the members of the plurality ofmembers; and determining a score for the electronic document, the scorebased on each rating value assigned to the electronic document by amember with each rating value being weighted by the weight factorassigned to the member who provided the rating.

In another aspect, a data processing system for assigning a score to anelectronic document is provided. The data processing system comprises:at least one processor; a memory operatively coupled to the at least oneprocessor; and a program module stored in the memory and operative forproviding instructions to the at least one processor, the at least oneprocessor responsive to the instructions of the program module. Theprogram module is operative for: assigning a weight factor to eachmember of a plurality of members; receiving a rating value of anelectronic document from each of at least two of the members of theplurality of members; and determining a score for the electronicdocument, the score based on each rating value assigned to theelectronic document by a member with each rating value being weighted bythe weight factor assigned to the member who provided the rating.

In another aspect, a memory for storing data for access by at least oneapplication program being executed on a data processing system isprovided. The memory comprises a data structure stored in said memory,said data structure including information resident in a database used bysaid at least one application program. The data structure comprising: adocument record associated with an electronic document and having ascore value; and at least two member records each member recordrepresenting one of a plurality of members, each member record having aweight factor and associated with the document record by a rating valueassigned to the electronic document by the member represented by themember record. Wherein the score value of the document record is basedon each rating value assigned to the electronic document associated withthe document record and with each rating value weighted by the weightfactor of the member record associated with the rating value.

In this manner, a conventional search engine can be used utilized tolocate relevant electronic documents and then any scores determined forthe located electronic documents can be displayed with the searchresults, allowing the user to see an assessment of the search resultsthat is independent from the search engine's ranking of the searchresults.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

While the invention is claimed in the concluding portions hereof,preferred embodiments are provided in the accompanying detaileddescription which may be best understood in conjunction with theaccompanying diagrams where like parts in each of the several diagramsare labeled with like numbers, and where:

FIG. 1 is schematic illustration of a conventional data processingsystem capable of implementing the methods of the present invention.

FIG. 2 is schematic illustration of a network configuration wherein adata processing system operative to implement the provided methods inaccordance with the present invention is connected over a network to aplurality of servers operating as a search engine.

FIG. 3 is a schematic illustration of a member record.

FIG. 4 is a schematic illustration of a rating record.

FIG. 5 is a flowchart of a method for a user to rate an electronicdocument the user has reviewed.

FIG. 6 is a flowchart of a method for a user to rate and categorize anelectronic document the user has reviewed

FIG. 7 is a screen shot of an exemplary browser for ranking a webpagebeing viewed.

FIG. 8 is a flowchart of a method for a user to rank another member inthe membership set.

FIG. 9 is a flowchart of a method for setting the weights of the variousmembers in a member set.

FIG. 10 is a flowchart of a method for calculating score values forelectronic documents based on the weighted ratings of the electronicdocuments;

FIG. 11 is a flowchart of a method of obtaining a set of search resultsand displaying the search results with any score values determined forthe electronic documents referenced by the search results; and

FIG. 12 is a schematic view of a screen shot displaying search resultsalong with score values for the electronic documents referenced by thesearch results.

DETAILED DESCRIPTION

FIG. 1 illustrates a data processing system 101 suitable for supportingthe operation of the present invention. The data processing system 101could be a personal computer, workstation, server, mobile computingdevice, cell phone, etc. The data processing system 101 typicallycomprises: at least one processing unit 103; a memory storage device104; at least one input device 105; a display device 106; a programmodule 108 and a network interface 110.

The processing unit 103 can be any processor that is typically known inthe art with the capacity to run the provided methods and is operativelycoupled to the memory storage device 4 through a system bus. In somecircumstances the data processing system 101 may contain more than oneprocessing unit 103. The memory storage device 104 is operative to storedata and can be any storage device that is known in the art, such as alocal hard-disk, etc. and can include local memory employed duringactual execution of the program code, bulk storage, and cache memoriesfor providing temporary storage. Additionally, the memory storage device104 can be a database that is external to the data processing system 101but operatively coupled to the data processing system 101. The inputdevice 105 can be any suitable device suitable for inputting data intothe data processing system 101, such as a keyboard, mouse or data portsuch as a network connection and is operatively coupled to theprocessing unit 103 and operative to allow the processing unit 103 toreceive information from the input device 105. The display device 106 isa CRT, LCD monitor, etc. operatively coupled to the data processingsystem 101 and operative to display information. The display device 106could be a stand-alone screen or if the data processing system 101 is amobile device, the display device 106 could be integrated into a casingcontaining the processing unit 103 and the memory storage device 104.The program module 108 is stored in the memory storage device 104 andoperative to provide instructions to processing unit 103 and theprocessing unit 103 is responsive to the instructions from the programmodule 108.

The network interface 110 allows the data processing system 101 to beconnected to a computer network such as an intranet or the internet.This network interface 110 could be an Ethernet card, modem or otherline based network system or a wireless connection such as CDPD,Bluetooth, 802.11, or other suitable network.

Although other internal components of the data processing system 101 arenot illustrated, it will be understood by those of ordinary skill in theart that only the components of the data processing system 101 necessaryfor an understanding of the present invention are illustrated and thatmany more components and interconnections between them are well knownand can be used.

Additionally, the invention can take the form of a computer readablemedium having recorded thereon statements and instructions for executionby a data processing system 101. For the purposes of this description, acomputer readable medium can be any apparatus that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.

Overview of the System

Data processing system 101 typically runs a browsing application, suchas Microsoft's Internet Explorer™, Mozilla Firefox™, Apple Safari™,Netscape Navigator™, Opera™ or other browser program. In one aspect, aplugin is used with the browsing application to implement some of thedisclosed methods.

FIG. 2 illustrates a network configuration wherein the data processingsystem 101 is connected over a network 255 to at least one search engineserver 250. The network 255 can comprise a single network such as theinternet or it could comprise a plurality of networks such as a wirelessnetwork, a wired network, etc. While in one aspect, the system issuitable for use with the Internet, it should be understood that thenetwork 255 could comprise other types of networks, such as an intranetor other suitable network.

A plurality of content servers 265 _(i) to 265 _(N) are configured toact as web servers and provide data and electronic content, generallyalthough not necessarily in the form of websites containing webpages, tothe data processing system 101. The data processing system 101 canaccess any of the content servers 265 to view electronic documents, suchas webpages, contained on the content servers 265. Typically, the dataprocessing system 101 uses a web browser application to access any ofthe content servers 265 _(i) to 265 _(N), which are web servers and theelectronic documents accessed on any of the content servers 265 aregenerally files in a markup language which the browser displays as a website and web pages on the data processing system 101.

A search engine server 250 is provided, configured to provide searchresult data to the data processing system 101. Well-known search enginesinclude Google™, Yahoo! Search™ MSN Search™, Ask.com™, but there aremany other search engines and many may be sufficient for search engineserver 250. Typically, a search query transmitted to the search engineserver 250 from the data processing system 101 is used by the searchengine server 250 to locate a plurality of electronic documents on thecontent servers 265. For example, the search results may be a list ofelectronic documents located by the search engine 250 that resides onthe content servers 265 with links to the content servers 265 to allow auser using the data processing system 101 to access the locatedelectronic document. Typically, the search engine server 250 accessesvarious search indexes that are populated with: links to electronicdocuments; meta-data describing the content of indexed electronicdocuments; or other meta-data and uses these electronic document indexesto construct a list of search results, as is known in the art. As iscommon in the art, these electronic document indexes are typicallypopulated using a web crawler.

Once the search engine server 250 has located a number of searchresults, a list of search results is passed back to the data processingsystem 101 to the user. Generally, although not necessarily; each searchresult in the list of search results comprises a link to an electronicdocument located on one of the content servers 265.

A central server 260 is provided and is operatively connected to a scoredatabase 270. The score database 270 contains a plurality of memberrecords 280 and a plurality of rating records 290.

A number of users (or members) form a member set that has access to thecentral server 260. Each user in the set of members will have a memberrecord 280 associated with him or her and stored in the score database270. In one aspect, member record 280 is in the format of a memberobject 300 as shown in FIG. 3. Each member object 300 corresponds withone of the members in the member set and is the record used by themember as they are accessing the central server 260. Each member object300 comprises: a user field 310 that contains an identifier (i.e. thename or a user name of the user) which identifies the specific user ormember that corresponds to the member object 300; a user password field320 that contains the password of the user or member; a weight field330, that indicates a weighting factor to be applied to any rating madeby the user; and an avatar field 340 that contains a link to a graphicthe member is using for his or her avatar.

Each member object 300 can be associated with one or more other memberobjects 300. This occurs when one member rates another member in themember set. The association is defined by a user rating object 350 thathas a user rating field 360 which contains the rating that the one userhas given to the other member. In this manner, a member can rate othermembers in the member set and this rating will become a factor in thatother member's rating contained in the weight field 330 of the memberobject 300 corresponding to the other member.

In addition to the score database 270 storing a number of member records280, the score database 270 also stores a number of document records290. In one aspect, document record 290 is in the format shown in FIG.4. The document record 290 contains a document object 400 associatedwith zero (0) or more document rating objects 405. Each document object400 corresponds to an electronic document. The electronic document couldbe a webpage, website, graphic image, sound recording movie recording orother electronic content located on a content server 265 and eachdocument rating object 405 associated with the document object 400corresponds, to a rating of the electronic document by a member in themember set.

Typically, each document object 400 comprises: a title field 410,identifying the title of the electronic document; a graphicalrepresentation 415 that provides a graphical representation of theelectronic document corresponding to the document object 400; an addressfield 420 that identifies the location of the electronic document on thecontent servers 265 typically using the uniform resource locator address(URL address) or the uniform resource identifier address (URI address);optionally, a category field 460, that identifies a category that theelectronic document has been classed in; and a score field 470, thatcontains a score value determined for the electronic document and basedon how members in the member set have rated the electronic document.

Additionally, each document object 400 is associated with one or moredocument rating objects 405 where each document rating object 405 holdsa rating assigned to the page or other piece of content by a member inthe member set. Each document rating object 405 typically comprises: adocument rating field 440, which contains a rating of the electronicdocument made by a member; a comment field 450, containing any comments(typically in a text string) that the member might have made regardingthe electronic content; and a date stamp 455, indicating the date and/ortime the member rated the electronic document.

The document rating field 440 is used to store a rating of theelectronic document by the member in the associated member object 300.This document rating field 440 reflects the associated member's opinionor judgment of the electronic document. The document rating field 440typically comprises a rating from 1 to 10 with 10 being the highest and1 being the lowest, however, any sufficient rating system could be used,such as numeric, alphanumeric (i.e. excellent, good, average, poor,horrible), symbolic (i.e. four stars, two thumbs up) or any othersuitable rating system.

The comment field 450 can hold a comment made by the associated memberregarding the electronic document identified in the address field 420 ofthe associated document object 400. The comment can be a long string oftext containing a comment about the site made by the creator.

The score field 470 in the document object 400 shows a rating of the webpage, of an electronic document, or other content identified by theaddress field 420. The rating contained in the score field 470 is basedon the rating contained in the content rating field 440 of all of theassociated document rating objects 405 with each rating being furtherweighted with the weight contained in the weight field 330 of the memberobject 300 associated with the member who provided the rating in thedocument rating field 440. In this manner. an electronic document can berated by a number of members with each rating assigned by a member beingfurther weighted based on how other members have personally rated thatmember.

It is to be understood that the member record data object 300 and thedocument object 400 and document rating object 405 are only illustrativeand that other formats with different fields may be used for the memberrecords 280 and document records 290 contained in the score database270.

Rating an Electronic Document

Each document record 290 stored in the score database 270 relates to arating a specific member has assigned to an electronic document whetherthe electronic document is a web site, web page or other item ofelectronic content on one of the content servers 265. As a user browseselectronic documents, the user can rate and comment on any of theelectronic documents they care to.

FIG. 5 illustrates a flowchart of a method 500 of a user rating anelectronic document the member has reviewed. In one aspect, method 500is implemented by a plug-in to a browser application on a dataprocessing system 101 being operated by the member. In another aspect,the method may be integrated into the browser application itself. Themethod 500 begins with a member logging into the central server 260 atstep 510. Typically, this involves the user entering a unique usernameand password into their data processing system 101 so that the dataprocessing system 101 can transmit this information to the centralserver 260. The user typically has previously registered with thecentral server 260 and a member record 280 has been createdcorresponding to the user making the user a member of the member set.The username and password that is transmitted to the central server 260is used to access the member record 280 in the score database 270 thatcorresponds to the user. This allows the central server 260 to determinewhich member from the set of members is accessing the central server260.

The user can then browse electronic documents on various content servers265 at step 520. The electronic documents browsed by the user arecontained on one of the content servers 265. The user can either findthe electronic document he or she is browsing by either accessing thecontent server 265 directly (such as by using a URL or a hyperlink toaccess the content on the content server 265) or by using a searchengine, such as the search engine server 250, as is commonly known inthe art, to locate the electronic content.

The user can continue to browse different electronic documents at step520 until he or she discovers an electronic document they would like torate. In one aspect, when a user rates a specific webpage on a site, theentire site is rated and associated with the rating. In another aspect,when the user rates a specific webpage in a site, the rating onlyapplies to the specific page and not the site itself.

The electronic document can be any electronic content that can beuniquely identified by an address (such as a URL address or URIaddress), i.e. a specific webpage, graphic, picture, file, link, etc.can be rated and/or a comment provided for by the user for thatelectronic document alternatively the electronic document can be awebsite that has a number of webpages all contained on a content server265.

The user then rates the electronic document at step 530. FIG. 7illustrates a screen shot on one embodiment of a web browser allowing auser to rate an electronic document that is a webpage being viewed bythe browser by selecting a drop down menu and selecting a rating from 0through 10, with 0 being the lowest and 10 being the highest.

Referring again to FIG. 5, the user can, optionally, also make a commentregarding the page or content at step 540 by submitting a typed comment.

The rating and optionally the comment is then forwarded to the centralserver at step 550. Referring to FIGS. 2 and 4, at the central server260, if a document object 400 does not exist that corresponds to theelectronic document being rated, a new document object 400 is createdand the title and URL address or URI address of the electronic documentbeing rated is placed in the title field 410 and address field 420 ofthe document object 400, respectively. The central server 260 uses thelog in information provided at step 510 to associate the new documentrating object 405 with the member object 300 of the user who rated theelectronic document. Additionally, a graphical representation of theelectronic content can be stored in the graphical representation field415. A document rating object 405 is created and associated with thedocument object 400 and the member object 300 corresponding to the userthat has provided the rating. The rating is then inserted in thedocument rating field 440 of the document rating object 405 and anycomment provided by the user is inserted in the comment field 450 of thedocument rating object 405. Optionally, a time and/or date the ratingwas made can be inserted in the date stamp field 455.

However, if a document object 400 that corresponds to the electronicdocument being rated already exists, then the existing document object400 corresponding to the electronic document being rated is obtained. Adocument rating object 405 is then created and the rating placed in thecontent rating field 440 of the new document rating object 405. Thecentral server 260 uses the log in information, provided at step 510, toassociate the new document rating object 405 with the member object 300of the user or member who rated the electronic document. If the userprovided a comment or review of the electronic document, the comment isinserted in the comment field 450. Typically, any field in the ratingrecord 290 that is empty is ignored when it is evaluated so if nocomment is provided, the comment field 450 is simply left empty.Optionally, a value indicating a time and/or date the rating was madecan be inserted in the date stamp field 455.

Referring again to FIG. 5, the method 500 ends and the user can log offthe central server 260 or alternatively go back to browsing web pages orother electronic content on the content servers.

In this manner, a user or member can rate a number of differentelectronic documents and a rating record 290 is created or updated bythe central server 260 in the score database 270. Each rating record 290indicates which members rated the electronic document and what thatrating was along with any comment about the electronic the user mighthave made. As more and more members of the member set rate electronicdocuments such as web pages and other electronic content, more and moreratings are associated with a specific electronic document. In thismanner, a large collection of ratings of an electronic document canaccumulated in the score database 270 with a number of different membersof the member set each subjectively rating each electronic document.

FIG. 6 illustrates a flowchart of a method 600 that varies slightly frommethod 500 illustrated in the flowchart in FIG. 5, in that it alsoallows a user to assign a category to the electronic document that he orshe is ranking. In one aspect, if a member ranks an electronic documentthat has not yet been rated by one of the members in the member set,method 600 is used instead of method 500 illustrated in FIG. 5. In thismethod, steps 510, 520, 530 and 540 are the same, however, an additionalstep 635 is added and step 655 is substituted in place of the previousstep 550 so that the member rating an electronic document for the firsttime can assign the electronic document a category of subject matterthat it falls within.

Step 635 has the user identify a category that the electronic documentthat is being ranked falls into. For example, the user may be presentedwith a list of categories they may choose from (i.e. sports, funny,games, technology, cooking, etc.) and the user may choose one of theprovided categories that he or she believes the electronic document mayfall into. Alternatively, step 635 may simply allow a user to enter atext string giving the user free reign to enter a category of theirdefinition. Additionally, the user may be presented with a mix of thesetwo options where he or she may be presented with a list of categoriesthey may choose from in addition to allowing them to add anothercategory if none of the presented categories seems sufficient to them.

At step 655 the information including the chosen category is provided tothe central server and the central server 260 adds the information tothe category field 460 of the document object 400 corresponding to theelectronic document being rated and the rating to the document ratingfield 440 of the newly created document rating object 405 along withassociating the newly created document rating object 405 with the memberobject 300 of the member that provided the rating.

Rating Other Members in the Member Set

Each member in the member set will rate electronic documents based ontheir own subjective opinions of the electronic document. Because eachmember rates the electronic documents subjectively, different users willoften give the same electronic document different ratings. In many casesthese ratings may be similar, but often different members may give thesame electronic document very different ratings. As the members of themember set get to know how specific members review and/or rate content,the other members can rate that member. A member can identify anothermember in the members set and provide a rating of this other member. Howa member of the member set is rated by other members will then affecthow their ratings of electronic documents is factored into the overallrating of the electronic documents.

FIG. 8 illustrates a flowchart of method 800 which allows a member torate another member in the member set. Method 800 comprises the stepsof: logging in 810; searching for a member 820; and rating the member830.

The method 800 begins and the user logs into the central server at step810. The log in typically involves the member providing a uniqueusername and password to the central server to allow the central serverto retrieve the member record from the database associated with themember.

At step 820, the member is able to search the list of members to locatea member they are interested in rating. Typically, the member is able toenter the name of the member in a search field and the central server260 will display possible matches of members that the member can thenselect from.

At step 830, the member can select another member in the member set andrate that selected member. Again, this rating can be of a number ofdifferent types, however, in one aspect it is again a rating on a scalefrom 0 to 10 with 0 being the lowest rank and 10 being the highest rank.Referring to FIG. 3, the rating is then stored in a user rating object350 in a user rating field 360. The user rating object 350 defines theassociation between the member providing the rating and the other memberhe or she is rating. This rating is later used to determine the weightin the weight field 330 of the member object 300.

In this manner, a member of the member set can rate a number of othermembers based on their previous comments and ratings of electronicdocuments or by their personal association with other members. Thisrating is stored in a user rating field 360 of a user rating object 350.For example, if a member likes the comments of another member and agreeswith his or her ratings of one or more electronic documents, the usermember may rare this member highly. Alternatively, a member may dislikeor disagree with another member's comments and/or rating of one or moreelectronic documents and therefore give that member a lower rating.

Determining the Weighted Ratings of the Members

In addition to each member being able to rate other members in themember set, each member's ratings of electronic documents are weightedby a weight factor based on how the other members in the member setrated that member. And not only are other member's subjective opinionsof a member's ratings and/or reviews used to provide that member withhis or her own weight factor that will affect that member's ratings ofelectronic documents, the weight factors determined for each of themembers can also be used to weight the member's ratings of other membersin the member set. In this manner, a rating of a member by a highlyrated member of the member set will have more effect on that member'sdetermined weight factor than a rating provided by a relatively lowlyrated member of the member set.

Each member in the member set will have a weight factor determined forhim or her that is based on the ratings of that member by other membersin the member set. These ratings by other members will in turn beweighted based on the rating member's own determined weight factor whichis in turn based on the weighted ratings of that member by othermembers. Referring to FIG. 3, the weight field 330 of a member's memberobject 300 contains the weight factor that has been determined for thatmember based on the weighted ratings of other members in the member set.Each rating of an individual member will be a based on the ratings givenby other members to that user (i.e. the ratings contained in the userrating field 360 of the user rating objects 350) weighted by the weightfactor determined for the rating member (i.e. the value contained in theweight field 330 of that rating member's member object 300).

FIG. 9 illustrates a flowchart of method 900 of setting the weightfactors of the various members in a member set. The method 900 comprisesthe steps of: selecting a first member object 910; obtaining all of theuser rating objects associated with a selected member object 920;determining the weight factor for a selected member object 930; checkingif any other member objects need to have a weight factor determined 940and if so, selecting another member object 950 and repeating steps 910,920, 930 for the next selected member object; iteratively determiningweight factors for the member objects until a sufficient approximationof the weight factors has been achieved 960; and ending the method 900.

At step 910 a first member object is selected and all of the user ratingobjects associated with the selected member object are obtained at step920. Referring to FIG. 3, each member object 300 is associated with anumber of user rating objects 350; a user rating object 350 for eachtime another member has rated that member. Each user rating object 350will be further associated with a member object 300 corresponding to themember that provided the rating.

Referring again to FIG. 9, at step 930, method 900 determines a weightfactor for the selected member object which will be stored in the weightfield. Referring to FIG. 3, the calculated weight factor is placed inthe weight field 330 of the selected member object 300. This weightfactor will be based on all of the ratings of the user by other membersin the member set (the values contained in the user rating field 360 ofthe user rating objects 350) with each rating in the user rating field360 weighted by the rating member's own determined weight factor (thevalue contained in the weight field 330 of the member object 300corresponding to the member doing the rating).

In one aspect, the weight factor of a user, w_(y), to be stored in theweight field 330 of the selected member object 300, where there are Nnumber of user ratings objects 350 associated with the member object300, is given by:

$w_{y} = \frac{\sum\limits_{x = 1}^{N}\; {f\left( {w_{x},r_{x}} \right)}}{\sum\limits_{x = 1}^{N}\; \left( w_{x} \right)}$

where w_(x) is the weight factor determined for a member that has ratedthe member (i.e. the value contained in the weight field 330 of themember object 300 for that member); r_(x) is the rating that the othermember gave to the user (i.e. the value contained in the user ratingfield 360 of the user rating object 350 associated with that member'smember object 300); and function ƒ (w_(x), r_(x)) is the rating given tothe user by a member weighted with the weight factor assigned to thatmember and taking into account the number of members who have rated theuser.

The specific equation used may be varied depending on the goals andvalues that are desired to be optimized by the equation, such as medianrating, whether the number of members who have rated the user should betaken into account, etc. However, in one aspect, using a rating systemof 1 through 10 the equation used to determine the weight factor, w_(y),could simply be the weighted mean as follows:

$w_{y} = \frac{\sum\limits_{x = 1}^{N}\; {w_{x}*r_{x}}}{\sum\limits_{x = 1}^{N}\; w_{x}}$

Referring again to FIG. 9, once a weight factor has been calculated forthe selected member object at step 930, the method 900 checks to see ifthere are more member objects to have their weight factors determined atstep 940. If weight factors need to be calculated for any more memberobjects, the next member object is selected at step 950 and steps 910,920, 930 and 940 are repeated until a weight factor has been determinedfor each of the member objects.

At step 960, iteration is used to repeatedly solve for the weight factorof the members in the member set. The member set forms a complex networkof inter-relations. Because weight factors of a member are a function ofboth the ratings of other members in the member set and of the weightfactors of the members providing the ratings, recursive relationshipsbetween the members are formed. When a new weight factor is calculatedor recalculated for one of the members, all of the weight factors ofthose members that have been rated by that member are also affected. Inthis manner, once a new weight factor is determined for a member in themember set, the weight factors of members rated by that member will beaffected and the weight factors of other members will also have to beupdated using that members newly determined weight factor. These membersrecalculated weight factors will also affect the weight factors of anyother members these members have rated making it desirable to updatethese other member's weight factors.

The steps of the method 900 are iteratively repeated until a sufficientlevel of convergence is reached. What is considered a sufficient levelof convergence will depend upon how precise the final calculated weightfactors of the members is desired.

The first time the method 900 is run by the central server 260 on themember records 280, a seeding value is used for the member weightfactors. Periodically, method 900 is run on the member objects 300 totake into account new ratings of members or changes to the ratings bymembers. Additionally, a member's rating of another member may have afinite period that is used before it is removed to prevent stale ratingsfrom affecting a user's rating unfairly. Referring to FIG. 3, the datestamp 370 of a user rating object 350 may be examined periodically toremove the user rating object 350 if a predetermined period of time haspassed since the rating was made.

Determining Scores for Rated Electronic Documents

Once weight factors have been determined for the members using method900 shown in FIG. 9, the ratings the members have assigned to electronicdocuments can be used along with the weight factors of the members whoprovided the ratings to determine a score for the electronic documents.

FIG. 10 illustrates a flowchart of a method 1000 for calculating a scorevalue for electronic documents rated by a member. The method 1000comprises the steps of: selecting a first document object 1010;obtaining all of the associated document rating objects for the selecteddocument object 1020; calculating a score for the document object 1030;checking if more electronic documents need to have scores calculated forthem 1040; selecting the next document object 1050 and repeating steps1020, 1030, 1040 and 1050 until scores for all of the document objectshave been determined.

Referring to FIGS. 4 and 10, at step 1010 a first document object 400representing an electronic document is selected and the document ratingobjects 405 associated with the selected document object 400 areobtained at step 1020.

A score for the document object is then calculated at step 1030. Thescore for each document object 400, which will be contained in the scorefield 470, is a function of the ratings of the electronic documentprovided by the members in the member set (the value in the documentrating field 440 of the document rating object 405) with each ratingweighted by the weight factor determined for the member making therating (the value in the weight field 330 of the member object 300associated with the document rating object 405).

In one aspect, the score, S, for an electronic document, y, is aweighted average of the ratings provided by members and given by theequation:

$S_{y} = \frac{\sum\limits_{x = 1}^{N}\; {w_{x}*r_{x}}}{\sum\limits_{x = 1}^{N}\; w_{x}}$

where r_(x) is the rating that the member gave to the page (i.e. thevalue contained in the document rating field 440 of the document ratingobject 405) and w_(x) is the weight factor determined for the memberthat has rated the page (i.e. the value contained in the weight field330 of the member object 300 for that member).

Once a score has been calculated for a selected document object 400 atstep 1030, method 1000 checks to determine if more electronic documentsneed to have a score determined for them at step 1040 and if there are,another document object 400 is selected at step 1050 and steps 1020,1030 and 1040 are repeated for the next document object 400.

Once all the document objects 400 are updated with scores, method 1000ends. In this manner, scores are calculated for electronic documentsthat have been rated by members in the member set.

Generally, method 1000 is repeated periodically in order to keep thescores for the electronic documents, that have been rated by members inthe member set, relatively up to date. Scores for electronic documentschange as members provide new ratings of the electronic document.Additionally, as members rate other members the weight factors of themembers who have rated the electronic documents change, causing thescores for the electronic documents these members have rated to change.Additionally, the ratings provided by members may be removed after aperiod of time to prevent stale ratings from affect the score determinedfor an electronic document. All of these factors will cause a score tochange and periodic updating is needed to keep the scores relatively upto date.

Displaying a List of Search Results Along with their Scores

The present method of rating an electronic document, such as a webpageor other piece of electronic content, can be used to enhance the displayof electronic documents. The electronic document can be displayed withthe score that was determined for it as describe herein, so that a userviewing the electronic document can also view the score determined forthe electronic document. The electronic document that is displayed doesnot necessarily have to be the complete document, but can simply be ameta-document describing the actual document, such as a search resultsreturned from a search engine and displayed in a list of search results,or with other meta-documents describing a number of different webpageswith a graphical representation of the webpage showing alongside thepage score and any comments made regarding the electronic document bymembers of the member set.

FIG. 11 illustrates a flowchart of a method 1100 for displaying a set ofsearch results along with any scores that have been determined forelectronic documents referenced by the search results. Method 1100comprises the steps of: receiving a search string 1110; requesting asearch 1120; obtaining a set of search results 1130; selecting a firstsearch results 1140; searching for a document object corresponding tothe selected search results 1150; checking if more search results remain1175; selecting the next search results 1180; repeating steps 1150, 1175and 1180 until no search results remain; and displaying the searchresults with any located scores 1185.

The method 1100 obtains a set of search results from a search engine,each search result in the set of search results referencing anelectronic document, such as webpages, website or other electroniccontent, that corresponds in some way to the search query used torequest the search. Any of the returned search results referencing anelectronic document that has had a score determined for it, has thescore displayed adjacent to the search result to indicate to a user whatscore the electronic document referenced by the search result received.As outlined above, the score determined for an electronic document isbased on ratings of the electronic document by members of a member setwith the ratings further weighted by a weight factor determined for themember doing the rating. The score can be determined using methods 500or 600 illustrated in FIGS. 5 and 6, respectively, to obtain ratings ofthe electronic document by members. User ratings of the members by othermembers can be obtained using method 800 illustrated in FIG. 8. Eachmember's user ratings can then be used as illustrated in method 900 inFIG. 9 to set a weight factor for each member and then the weight factordetermined for each member using method 900 can be used as with method1000 as illustrated in FIG. 10 to obtain a score for the electronicdocument. The score for an electronic document is then displayedadjacent to a search result referencing the electronic document toprovide a user that requested the search with a rating of locatedelectronic documents that is independent from the ranking of theelectronic documents assigned to the electronic document by the searchengine.

Method 1100 begins and at step 1110 when a search query is received.Typically, a user inputs the search query that they would like used tosearch for related electronic documents into the data processing system101. As is commonly known in the art, the search query typicallyconsists of one or more words or search terms that the user believes arerelated to the contents of the electronic documents he or she isinterested in. Typically, the search query can also contain operatorssuch as “and”, “or”, “not”, etc. that are used to indicate therelationship between the words or search terms in the search query.

At step 1120 a search is requested using the search query. Referring toFIG. 2 the data processing system 101 sends a request containing thesearch string to the search engine server 250. As is commonly known inthe art, the search engine server 250 searches its search index forelectronic document (typically webpages) that correspond to the searchquery based on the searching algorithm used by the search engine server250. The search engine server 250 then determines a set of searchresults, where each search result in the set of search resultsreferences an electronic document that has been located in the searchindex using the search query. The set of search results is returned asan ordered list of search results. Each search result is ranked by thesearch engine server 250 using a ranking algorithm and the rankdetermined for each search result determines the order the searchresults in the list of search results.

Each search result in the set of search results references an electronicdocument. For example, if the electronic document is a webpage, thesearch result does not provide the actually webpage, but merelyreferences the webpage allowing a user to access the webpage if theychose the search result. Each search result typically contains metadatadescribing the contents of the electronic document along with a title ofthe electronic document and a link to access the electronic document,such as a URL address or URI address indicating the location ofelectronic document on the internet.

Referring again to FIG. 11, at step 1130 the data processing system 101receives the set of search results from the search engine server 250.

At step 1140 the method 1100 selects a first search result from the setof search results and at step 1150 the method 1100 checks whether ascore has been determined for the electronic document referenced by theselected search result. Referring to FIG. 2, the method 1100 accessesthe central server 260 which in turn accesses the score database 270 andthe document records 290 contained in the database 270. Referring toFIG. 4, the document records 290 are checked to see if any of thedocument objects 400 correspond to the electronic document referenced bythe selected search result. Typically, the URL address or URI addresscontained in the selected search result is transmitted to the centralserver 260 to be used to compare to the URL addresses or URI addressesstored in the address fields 420 of the document objects 400 todetermine if any of the document objects 400 correspond to the selectedsearch results. If one of the document objects 400 contained in thescore database 270 matches the selected search results, then any scorecontained in the score field 470 of the matching document object 400 isobtained and associated with the search result. The score in the scorefield 470 is sent back to the data processing system 101 running themethod 1100.

Referring again to FIG. 11, at step 1175 the method 1100 checks todetermine whether there are more search results remaining in the set ofsearch results returned from the search engine server 250 and if thereare more search results to be examined, the method 1100 selects the nextsearch result at step 1180 and steps 1150, 1175 and 1180 are repeated aslong as there are search results remaining in the set of search results.Each search result is checked at step 1150 for a corresponding documentobject 400 and the score obtained for any search results referencing anelectronic document that has had a score determined for it.

At step 1175 when there are no more search results remaining in the setof search results, each search result referencing an electronic documentthat has had a score determined for it has obtained the score, method1100 moves on to step 1185 and the set of search results are displayedalong with any located scores. The set of search results are typicallydisplayed in a list (as is commonly known in the art) with the searchresults in the order provided by the search engine 250 with any scoreassociated with a search result positioned adjacent the search result sothat a user can identify which scores are related to which searchresults.

Displaying the Search Results with Located Scores

FIG. 12 is a schematic illustration of a screen shot of an interface1200 displaying a list of search results 1205 along with any score in ascore indicator 1230 that has been determined for the search results1210 in one aspect of step 1185 of method 1100 shown in FIG. 11.

A search query input field 1208 is provided at the top of the interface1220 to allow a user to input a new search query and request a newsearch to be conducted using the new search query.

A list of search results 1205 from a previously requested searchreturned from a search engine is provided. The list of search results1205 contains a number of search results 1210 with a score valueindicator 1230 positioned adjacent to any search result 1210 that hashad a score value determined for them. Each search result 1210references an electronic document such as a webpage or other electroniccontent that was located in the search. As is commonly known in the art,a user can select any of the search results 1210 (typically using amouse and pointer to click on the search result 1210) in order to accessthe electronic document referenced by the search results 1210. The userselecting a search result 1210 usually results in the user navigating tothe electronic document referenced by the selected search result 1210.Typically, the list of search results 1205 is longer than a singlescreen so a scroll bar 1250 is typically provided to allow a user toscroll through the list of search results 1205.

Each search result 1210 in the list of search results 1205 typically hasa title 1212, metadata 1214 and a URL address 1216. The title 1212displays the title of the electronic document referenced by the searchresult 1210. For example, if the electronic document is a webpage, thetitle 1212 displays the title of the webpage. If the electronic documentis an image or graphic, the title 1212 can display a title for the imageor movie or the filename of the image or movie. The metadata 1214typically displays information about the electronic document referencedby the search result 1210. The information typically describes thecontents of the electronic document to the user to allow the user todecide whether they would like to access the electronic documentreferenced by the search result 1210. If the electronic document is awebpage, the metadata 1214 can be a metatag for the webpage providing asummary of the contents of the webpage. The address 1216 provides thelocation of the electronic document, so that the electronic document canbe accessed. Typically, the address 1216 indicates the URL address ofthe located electronic document.

The list of search results 1205 is ordered and each search result 1210in the list of search results 1205 is displayed in the order which wasdetermined by the search engine server 250 that provided the list ofsearch results 1205. The order of the search results 1210 in the list ofsearch results 1205 is based on the ranking of the electronic documentsreferenced by the search result 1210 that was determined by the searchengine server 250.

For any search result 1210 referencing an electronic document for whicha score has been determined, a score indicator 1230 is displayed withthe search result 1210. The score indicator 1230 displays the scoredetermined for the electronic document referenced by the adjacent searchresult 1210. The scores in the score indicators 1230 for the differentsearch results 1210 are obtained using method 1100 illustrated in FIG.11. As discussed above, the score for the electronic document is basedon user ratings assigned to the electronic document made by the membersof a member set with the user rating assigned to the electronic documentfurther weighted by a weight factor associated with the member providingthe user rating. In FIG. 12, the score is a number in a scale from oneto ten (10), however, a different scoring value system could also beused. In FIG. 12, each score indicator 1230 is displayed adjacent to theassociated search result 1210 with the score indicator 1230 being placedon one side, however, the score indicator 1230 could be displayed to anyside of the associated search results 1210 or merged within the contentsof the search result 1210.

For search results 1210 where no score was determined or located for thereferenced electronic document, a null indication 1240 can be shownindicating that the search result 1210 does not have a score associatedwith it. In FIG. 12, an “NR” or “not rated” is shown to indicate that noscore was located for the electronic document referenced by a searchresult 1210, however, any number of different indicators could be usedto inform a user that there is no corresponding score for the referencedelectronic document.

In this manner, a user can request a search for electronic documents,such as webpages, from a search engine server 250 such as Google™,Yahoo! Search™, Excite™, etc. As is known in the art, a list of searchresults 1205 will be returned from the search engine server 250 witheach search result 1210 referencing a webpage or other electronicdocument located on the Internet. Each search result 1210 in the list ofsearch results 1205 will have been ranked and ordered by the searchengine server 250 before being sent to the data processing system 101.As is known in the art, the ranking and order determined by the searchengine server 250 will depend upon the ranking algorithm employed by thesearch engine server 250. The webpage or electronic document is rankedby the search engine based on a number of different criteria. Forexample, if the electronic document is a webpage, typically one of thesecriteria used by the ranking algorithm of the ranking by the searchengine server 250 is based on the number of occurrences of the searchterms in the metatag of the webpage or the content of the webpageitself. Another criterion may typically be whether the domain name ofthe webpage contains search terms from the search string. Any number ofother criteria could also be used including sponsorship of the search,i.e. the owner of the webpage paying a fee to the search engine to ranktheir webpage higher. The search results 1210 are ordered in the list ofsearch results 1205 based on the ranking of the search results 1210 bythe search engine server 250. In the interface 1200, the originalordering of the list of search results 1205 by the search engine server250 is maintained. Any score determined for an electronic documentreferenced by a search result 1210 is displayed with the search result1210 in a score indicator 1230.

In this manner, a user can see how other people have rated the searchresult 1210, with their ratings being weighted based on how the personproviding the rating was rated by other people. Not only does a user getto see how the search engine server 250 ranks the search results 1210based on the order returned, the user also gets to see a scoredetermined for the search results 1210 that is independent of theranking algorithm used by the search engine server 250 and rather thanbeing based on some objective measurements of the electronic document,the score is based on subjective ratings of the electronic documentprovided by people, all while utilizing a third party search engineserver 250 to obtain search results.

Searching by Category

The present method of rating an electronic document, such as a webpageor other piece of electronic content, can be used be used to rank acollection of electronic documents. A collection of stored electronicdocuments can be ranked using the score determined as described hereinand the electronic documents displayed in an order based on theirdetermined scores or based on a ranking formula that factors in thescores as part of the ranking formula. Additionally, by having membersin the member set assign a category to an electronic document that theyhave rated. the electronic documents can be sorted by category, with allthe electronic documents assigned to one category grouped together andthen ranked within the category using the score. This allows users tosearch an electronic document collection by selecting a category ofelectronic documents they are interested in and then viewing the rankedelectronic documents.

As a Ranking System or as a Component of a Ranking System in a SearchEngine

The present method of determining a score for rating an electronicdocument, such as a webpage or other piece of content. based on weightedmember ratings from members in a member set can be used in conjunctionwith a search engine's search index. The rating method can be used torank electronic documents located in the search index or as a factor ina larger ranking algorithm used by a search engine to rank the resultsof a search of its index or database. The rating method is incorporatedinto a web search engine to rank located electronic documents takinginto account the ratings. The search engine will locate electronicdocuments that match a search criteria and generate a list of electronicdocuments. This list can then be sorted by the search engine using thescore calculated for the electronic documents using the rating methoddisclosed above with the high ranking electronic documents listed firstand lower ranking electronic documents listed later. The rankings can beaccomplished using solely the score determined using the rating methodsdisclosed or with the score as one of a number of factors as part of alarger ranking algorithm.

Reordering List Based On

The present method of assigning a score to a document can also be usedto reorder a list of search results. A search conducted by a third partysearch engine or other search service typically returns a list of searchresults that have been ranked by the search engine or other searchservice itself. By using the score determined as described herein theselists of search results can be reordered by re-ranking search resultsthat have been rated by members in the member set.

A user can also reorder the list of search results 1205 so that thesearch results are reordered based on the score indicator 1230. The listof search results 1205 will be reordered based on the score indicated inthe score indicator 1230. In one aspect, reorder button 1260 can beselected by a user to reorder the list of search results 1205 based onthe score shown in the score indicator 1230.

The foregoing is considered as illustrative only of the principles ofthe invention. Further, since numerous changes and modifications willreadily occur to those skilled in the art, it is not desired to limitthe invention to the exact construction and operation shown anddescribed, and accordingly, all such suitable changes or modificationsin structure or operation which may be resorted to are intended to fallwithin the scope of the claimed invention.

What is claimed is:
 1. A method for displaying a plurality of scores,the method comprising: deploying a network-connected electronic documentscoring server comprising at least a memory and a processor and furthercomprising programmable instructions stored in the memory and operatingon the processor, the instructions configured to displaying a pluralityof scored document objects, comprising the steps of: receiving aplurality of document objects; displaying, at a display device, at leastone score along with the plurality of document objects, each scoreassociated with each document object; wherein each score is determinedindependently from the received order of the plurality of documentobjects; wherein each score is based on at least one rating valuesubjectively assigned to the associated document object by a firstmember of a plurality of members other than the first member; and,wherein each rating value is weighted by a weight factor assigned to thefirst member who assigned the rating value, with the weight factorassigned to the first member being based on user ratings assigned to thefirst member by other members of the plurality of members, where eachuser rating used to determine the weight factor for a second member isweighted by the weight factor determined for the second member assigningthe user rating and wherein the weight factors for the plurality ofmembers are determined by iteration before the at least one score of theplurality of scores is determined.
 2. The method of claim 1, whereineach rating value used to determine one of the at least one score isweighted by a weight factor assigned to the first member who assignedthe rating value.
 3. The method of claim 1, wherein the plurality ofdocument objects are links to webpages.
 4. The method of claim 1,wherein the plurality of document objects are references to electronicdocuments.
 5. The method of claim 1, further comprising the step ofreordering the plurality of document objects based on the at least onescore.
 6. A document scoring system comprising: at least one processor;a memory operatively coupled to the at least one processor; a displaydevice operative to display data; and program code stored in the memoryand operating on the at least one processor, the instructions configuredto displaying a plurality of scored document objects, the program codeoperative for: receiving a plurality of document objects; displaying, ata display device, at least one score along with the plurality ofdocument objects, each score associated with each document object;wherein each score is determined independently from the received orderof the plurality of document objects; wherein each score is based on atleast one rating value subjectively assigned to the associated documentobject by a first member of a plurality of members other than the firstmember; and, wherein each rating value is weighted by a weight factorassigned to the first member who assigned the rating value, with theweight factor assigned to the first member being based on user ratingsassigned to the first member by other members of the plurality ofmembers, where each user rating used to determine the weight factor fora second member is weighted by the weight factor determined for thesecond member assigning the user rating and wherein the weight factorsfor the plurality of members are determined by iteration before the atleast one score of the plurality of scores is determined.
 7. The scoringsystem of claim 6, wherein each rating value used to determine one ofthe at least one score value is weighted by a weight factor assigned tothe first member who assigned the rating value.
 8. The method of claim6, wherein the plurality of document objects are links to webpages. 9.The method of claim 6, wherein the plurality of document objects arereferences to electronic documents.
 10. The method of claim 6, furthercomprising the step of reordering the plurality of document objectsbased on the at least one score.
 11. A method for associating scoresdetermined for a plurality of documents objects, the method comprising:deploying a network-connected electronic document scoring servercomprising at least a memory and a processor and further comprisingprogrammable instructions stored in the memory and operating on theprocessor, the instructions configured to displaying a plurality ofscored document objects, comprising the steps of: providing a databasefor storing scores determined for a plurality of stored documentobjects; obtaining the plurality of stored document objects; obtainingone or more referenced document objects; for each referenced documentobject of the plurality of the plurality of referenced document objects,searching the database for any score determined for the correspondingstored document object wherein a referenced document object is the sameas corresponding stored document object; if a score has been determinedfor a first stored document object wherein the first document object isthe same as a first referenced document object, retrieving the score;wherein each score is based on at least one rating value subjectivelyassigned to the associated stored document object by a first member of aplurality of members; wherein each rating value is weighted by a weightfactor assigned to the member who assigned the rating value, with theweight factor assigned to a member being based on user ratings assignedto the member by other members of the plurality of members, where eachuser rating used to determine the weight factor for a member is weightedby the weight factor determined for the member assigning the user ratingand wherein the weight factors for members are determined by iterationbefore the at least one score is determined; and wherein the at leastone score is determined and associated to the first stored documentobject before obtaining the one or more referenced document objects. 12.The method of claim 11, wherein each rating value used to determine oneof the at least one scores is weighted by a weight factor assigned tothe member who assigned the rating value.
 13. The method of claim 11,wherein the plurality of document objects are links to webpages.
 14. Themethod of claim 11, wherein the plurality of document objects arereferences to electronic documents.
 15. The method of claim 11, furthercomprising the step of reordering the plurality of document objectsbased on the at least one score.